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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1 ,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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Abstract 

Extensive  cost  overruns  in  major  defense  programs  are  common,  and  studies  have  identified 
poor  cost  estimation  as  a  main  contributor.  Research  and  experience  have  identified  several 
factors  associated  with  poor  cost  estimates.  These  include 

•  optimistic  expectations  about  the  program  scope  and  technology  that  can  be 
delivered  on  schedule  and  within  budget; 

•  the  enormous  amount  of  unknowns  and  uncertainty  that  exist  when  these 
estimates  are  made  about  large-scale,  unprecedented  systems  that  take  years  to 
develop  and  deploy;  and 

•  the  heavy  reliance,  of  necessity,  on  expert  judgment. 

In  this  paper,  we  describe  a  new,  integrative  approach  for  pre-Milestone  A  cost  estimation 
called  quantifying  uncertainty  in  early  life  cycle  cost  estimation  (QUELCE).  QUELCE 
synthesizes  scenario  building,  Bayesian  belief  network  (BBN)  modeling,  and  Monte  Carlo 
simulation  into  an  estimation  method  that  quantifies  uncertainties,  allows  subjective  inputs, 
visually  depicts  influential  relationships  among  change  drivers  and  outputs,  and  assists  with 
explicit  description  and  documentation  underlying  an  estimate.  We  use  scenario  analysis  and 
dependency  structure  matrix  (DSM)  techniques  to  limit  the  combinatorial  effects  of  multiple 
interacting  program  change  drivers  to  make  modeling  and  analysis  more  tractable. 

Finally,  we  describe  results  and  insights  gained  from  applying  the  method  retrospectively  to  a 
major  defense  program. 

Background 

The  inaccuracy  of  cost  estimates  for  developing  major  Department  of  Defense  (DoD) 
systems  is  well  documented,  and  cost  overruns  have  been  a  common  problem  that 
continues  to  worsen  (GAO,  2011, 2012).  Because  estimates  are  now  prepared  much  earlier 
in  the  acquisition  life  cycle,  well  before  concrete  technical  information  is  available,  they  are 
subject  to  greater  uncertainty  than  they  have  been  in  the  past  (RAND,  2007).  Early  life 
cycle  cost  estimates  are  often  based  on  a  desired  capability  rather  than  a  concrete  solution. 
Faced  with  investment  decisions  based  primarily  on  capability,  several  problems  emerge 
when  creating  estimates  at  this  early  stage  (Roper,  2010): 

•  Limited  Input  Data:  The  required  system  performance,  the  desired 
architecture  of  the  solution,  and  the  capability  of  the  vendors  are  not  fully 
understood. 

•  Uncertainties  in  Analogy-Based  Estimates:  Most  early  estimates  are  based 
on  analogies  to  existing  products.  While  many  factors  may  be  similar,  the 
execution  of  the  program  and  the  technology  used  as  part  of  the  system  or  to 
develop  it  are  often  different.  For  example,  software  product  size  depends 
heavily  on  the  implementation  technology,  and  the  technology  heavily 
influences  development  productivity.  Size  and  productivity  are  key 
parameters  for  cost  estimation. 

•  Challenges  in  Expert  Judgment:  Wide  variation  in  judgment  can  exist 
between  experts,  and  the  confidence  in  the  input  that  they  provide  is 
generally  not  quantified  and  unknown. 
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•  Unknown  Technology  Readiness:  Technology  readiness  may  not  be  well 
understood,  and  is  likely  to  be  over-  or  underestimated. 

This  paper  describes  the  QUELCE  method  and  experiences  to  date. 

An  Improved  Method  for  Early  Life  Cycle  Cost  Estimation 

The  quantifying  uncertainty  in  early  life  cycle  cost  estimation  (QUELCE)  method  is  an 
integrative  approach  for  pre-Milestone  A  cost  estimation  to  address  the  problems 
associated  with  early  life  cycle  cost  estimation  while  at  the  same  time  providing  benefits  not 
found  in  current  cost  estimation  methods  (Ferguson  et  al.,  2011).  The  method  aims  to 
provide  credible  program  cost  estimates  as  distributions  rather  than  point  estimates. 
QUELCE  produces  intuitive  visual  representations  of  the  data  that  explicitly  model  influential 
relationships  and  interdependencies  among  the  drivers  on  which  the  estimates  depend. 
Assumptions  and  constraints  underlying  the  estimates  are  well  documented,  which 
contributes  to  better  management  of  cost,  schedule,  and  adjustments  to  program  scope  as 
more  is  learned  and  conditions  change.  Documenting  the  basis  of  an  estimate  facilitates 
updating  the  estimate  during  program  execution  and  helps  others  make  informed  judgments 
about  estimation  accuracy. 

The  QUELCE  method  differs  from  existing  methods  because  it 

•  uses  available  information  not  normally  employed  for  program  cost 
estimation, 

•  explicitly  models  uncertainty  on  the  input  side  of  the  cost  estimation  equation 
in  terms  of  program  change  drivers, 

•  enables  calculation  (and  re-calculation)  of  the  cost  impacts  caused  by 
changes  that  may  occur  during  the  program  life  cycle,  and 

•  enhances  decision-making  through  the  transparency  of  the  assumptions 
going  into  the  cost  estimate. 

Figure  1  shows  the  flow  of  information  in  a  typical  major  defense  acquisition  program 
(MDAP)  acquisition,  with  blue  boxes  added  to  represent  the  contributions  from  the  QUELCE 
method. 
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Figure  1.  Information  Flow  for  Early  Life  Cycle  Estimation,  With  QUELCE 

Method  Additions 


The  QUELCE  Method 

QUELCE  synthesizes  scenario  building,  Bayesian  belief  network  (BBN)  modeling, 
and  Monte  Carlo  simulation  into  an  estimation  method  that  quantifies  uncertainties,  allows 
subjective  inputs,  visually  depicts  influential  relationships  among  change  drivers  and 
outputs,  and  assists  with  the  explicit  description  and  documentation  underlying  an  estimate. 

It  uses  scenario  analysis  and  dependency  structure  matrix  (DSM;  Lindemann,  n.d.) 
techniques  to  eliminate  cycling  among  the  interacting  program  change  drivers  to  make 
modeling  and  analysis  more  tractable.  Representing  scenarios  as  BBNs  enables  sensitivity 
analysis,  exploration  of  alternatives,  and  quantification  of  uncertainty. 

The  BBNs  and  Monte  Carlo  simulation  are  used  to  predict  variability  of  what  become 
the  inputs  to  the  existing  cost  estimation  models  and  tools.  As  a  result,  interim  and  final  cost 
estimates  are  represented  as  distributions  so  that  the  decision-maker  can  see  the  probability 
of  a  program  exceeding  the  specified  cost.  The  method  can  be  described  as  a  series  of  five 
activities,  summarized  in  the  following  sections.2 

Identify  Program  Change  Drivers 

The  identification  of  program  change  drivers  is  best  accomplished  by  the  experts 
who  provide  programs  with  information  about  acquisition,  development,  and  the  technical 
approach,  in  addition  to  direct  input  for  cost  estimation.  A  workshop  setting  is  used  to 
identify  drivers  that  could  affect  program  costs.  These  experts  consider  all  aspects  of  a 
program  that  might  change  and  significantly  affect  its  execution  during  the  program’s  life 
cycle — particularly  given  the  new  information  developed  during  the  Technology 
Development  Phase  in  preparation  for  Milestone  B.  The  probability  of  program  success 
(POPS)  factors  used  by  the  Navy  and  Air  Force  can  be  used  to  start  the  brainstorming  and 
discussion. 


2  This  work  was  originally  described  in  a  two-part  series  on  the  SEI  blog,  A  New  Approach  for 
Developing  Cost  Estimates  in  Software  Reliant  Systems  (http://blog.sei.cmu.edu/post.cfm/improving- 
the-accuracy-of-early-cost-estimates-for-software-reliant-systems-first-in-a-two-part-series). 
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In  support  of  this  step,  we  have  found  that  there  is  much  useful  information  contained 
in  a  variety  of  documents  produced  during  the  pre-Milestone  A  phase.  These  include  the 
Analysis  of  Alternatives  and  the  various  reports  and  documents  developed  as  part  of  the 
Materiel  Solution,  the  Technology  Development  Strategy,  and,  where  available,  any  pre- 
Milestone  A  assessments  such  as  the  POPS  gate  reviews.  While  these  traditionally  have 
not  been  considered  for  cost  estimation  purposes,  during  the  conduct  of  a  retrospective 
study,  we  found  these  and  other  program  documents  to  contain  relevant  information 
suggesting  several  program  change  factors.  Our  initial  list  totaled  nearly  60  factors. 

In  the  workshops,  experts  are  asked  to  provide  judgments  about  the  status  of  each 
program  change  driver.  The  specific,  assumed  state  as  proposed  by  the  Materiel  Solution 
and  Technology  Development  Strategy  is  identified  and  labeled  as  the  nominal  state. 

Experts  then  brainstorm  about  possible  changes  in  the  condition  of  each  driver  that  may 
occur  during  the  program  life  cycle.  The  experts  identify  possible  changes  that  might  occur 
to  the  nominal  state  and  use  their  best  judgment  for  the  probability  that  the  nominal  state  will 
change. 

Identify  Interdependencies  and  Reduce  Complexity 

Once  the  changed  conditions — referred  to  as  potential  driver  states — are  fully 
identified,  participants  subjectively  evaluate  the  cause  and  effect  relationships  among  the 
drivers.  Expert  judgment  is  applied  to  rank  the  causal  effects.  A  matrix  is  developed  that 
provides  the  relationship  between  nominal  and  dependent  states  and  contains  the 
conditional  probability  that  one  will  affect  the  other,  but  not  the  impact  of  the  change.  This 
exercise  can  result  in  a  very  large  number  of  program  change  drivers  and  states  identified 
for  an  MDAP. 

Using  dependency  structure  matrix  (DSM;  Lindemann,  n.d.)  techniques,  the  highly 
rated  change  drivers  in  the  matrix  can  be  reduced  to  an  efficient  set  that  has  the  most 
potential  impact  to  program  execution  and,  hence,  cost.  The  DSM  technique  is  a  well- 
established  method  to  reduce  complicated  dependency  structures  to  a  manageable  size. 
Furthermore,  the  technique  helps  to  eliminate  cycles  in  the  matrix  by  transforming  the  matrix 
to  an  upper-right  triangle  and  makes  it  directly  useful  for  constructing  the  BBN.  An  example 
of  a  dependency  matrix  after  DSM  transformation  created  during  an  SEI  workshop  is 
provided  in  Figure  2. 
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Figure  2.  Example  Dependency  Matrix  After  DSM  Transofrmation 


Construct  a  Bayesian  Belief  Network 

A  BBN  is  constructed  using  the  program  change  drivers  derived  from  the  expert 
workshop  and  their  cause-and-effect  relationships.  The  BBN  models  the  change  drivers  as 
nodes  in  a  quantitative  network  and  includes  the  conditional  probabilities  that  changes  of 
state  in  one  node  will  create  a  change  of  state  in  another  node,  as  envisioned  by  the 
program  domain  experts.  Figure  3  depicts  an  abbreviated  visualization  of  a  BBN,  with 
circled  nodes  representing  program  change  drivers  and  arrows  representing  either  cause- 
and-effect  relationships  or  leading  indicator  relationships.  This  example  shows  that  a 
change  in  the  Mission  &  CONOPS  driver  will  likely  cause  a  change  to  the  Capability 
Analysis  driver,  which  in  turn  will  likely  change  the  Key  Performance  Parameters  (KPPs) 
driver  and  subsequently  the  Technical  Challenge  outcome  factor.  The  three  outcome  factors 
(Product  Challenge,  Project  Challenge,  and  Size  Growth)  and  their  corresponding  states  are 
mapped  to  some  of  the  traditional  cost  model  input  factors  and  their  values. 
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Conditional  probabilities  are  assigned  to  the  nodes  (drivers)  in  the  BBN.  Each  node 
can  assume  a  variety  of  states  with  an  associated  likelihood  identified  by  the  domain 
experts.  This  allows  the  calculation  of  outcome  distributions  on  the  variables. 


Domain  experts  use  the  BBN  to  define  scenarios.  The  realization  of  a  potential  state 
in  a  particular  node  is  specified,  and  the  cascading  impacts  to  other  nodes  and  the  resulting 
change  in  the  outcome  variables  are  recalculated.  Any  change  in  one  or  more  nodes 
(drivers)  constitutes  a  scenario.  Once  the  experts  are  satisfied  that  a  sufficient  number  of 
scenarios  are  specified,  they  use  their  judgment  to  rank  them  for  likely  impacts  to  cost.  An 
example  scenario  created  during  an  SEI  pilot  workshop  is  provided  in  Figure  4. 
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Figure  4.  A  Partial  Example  of  a  Scenario  With  Two  Driver  Nodes  in  a  Nominal 

State 


Select  Cost  Estimating  Models  to  Generate  an  Estimate 

Parametric  cost  estimation  models  for  software  use  a  mathematical  equation  to 
calculate  effort  and  schedule  from  estimates  of  size  and  a  number  of  parameters.  A 
decision  is  made  as  to  which  cost  estimating  tools,  cost  estimating  relationships  (CERs),  or 
other  methods  will  be  used  to  form  the  cost  estimate.  COCOMO  II  is  a  well-known 
estimation  tool  and  is  open  source.  The  SEI  has  so  far  developed  the  relationships  between 
BBN-modeled  program  change  drivers  and  COCOMO,  shown  in  Figure  5.  The  red  X’s  in 
brackets  indicate  an  inverse  relationship  between  the  BBN  output  factor  and  the 
corresponding  COCOMO  II  driver.  The  black  X’s  indicate  a  positive  relationship.  The  BBN 
interface  to  the  commercial  SEER-SEM  cost  estimating  tool  is  currently  underway. 
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Figure  5.  Mapping  BBN  Outputs  to  COCOMO  Inputs 


The  program  office  estimates  of  size  and  other  cost  model  inputs  such  as 
productivity  are  used  as  the  starting  point  in  this  step.  Often  these  values  are  estimated  by 
analogy  and  aggregation.  They  are  adjusted  by  applying  the  distributions  calculated  by  the 
BBN. 

Monte  Carlo  Simulation 

From  each  selected  scenario,  we  use  the  output  of  the  BBN  to  parameterize  a  Monte 
Carlo  simulation  of  the  inputs  to  the  selected  cost  estimation  model.  This  provides 
probability  distributions  for  the  input  factors  to  the  cost  estimating  models.  This  also  provides 
explicit  confidence  levels  for  the  results.  Figure  6  shows  the  simulation  results  that  the  SEI 
obtained  when  modeling  a  factor  (person-months)  in  three  different  scenarios. 
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Figure  6.  Simulation  Results  for  Three  Scenarios 

A  report  with  the  final  cost  estimates  is  generated  for  each  scenario,  including  the 
nominal  (expected)  program  plan.  The  explicit  confidence  levels  and  the  visibility  of  all 
considered  program  change  drivers  allow  for  quick  comparisons  and  future  re-calculations. 
This  method  enables  the  creation  of  comparative  scenario  calculations  at  any  point  during 
the  life  cycle.  The  visibility  of  the  program  change  drivers  and  the  transparency  afforded  by 
the  consideration  of  alternative  scenarios — and  their  assumptions — enables  improved 
decision-making  and  contingency  planning. 


Results  and  Future  Research 

To  date,  there  have  been  two  empirical  thrusts  to  the  research.  First,  we  have 
conducted  a  retrospective  on  an  MDAP.  We  constructed  a  10-year  time  line  of  the  program 
using  archival  documents,  records  from  various  DoD  repositories,  and  collaborations  with 
SEI  staff  who  worked  on  the  program. 


The  team  accessed  over  4, 100  program  files,  which  documented  virtually  all  of  the 
program’s  history.  In  addition,  the  team  obtained  over  100  official  contractor  submissions  of 
Software  Resource  Data  Reports  (SRDRs)  and  Earned  Value  Management  Reports 
contained  in  the  Defense  Automated  Cost  Information  System.  We  also  obtained  acquisition 
reports  from  the  Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  Purview 
repository,  which  included  the  relevant  Selected  Acquisition  Reports  (SARs)  and  the 
Defense  Acquisition  Executive  Summary  (DAES)  Reports. 
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With  the  participation  of  two  in-house  experts  who  had  worked  with  the  program,  we 
established  a  provisional  set  of  57  program-specific  change  drivers.  We  also  elicited  their 
judgment  on  the  likelihood  of  change  for  each  of  the  program  change  drivers  and  their 
potential  cascading  effect  on  the  other  drivers.  These  judgments  formed  the  basis  for 
implementing  DSM  techniques  to  reduce  the  complexity  and  capture  the  cascading  effects 
of  the  interdependencies  among  the  program  change  drivers. 

DSM  reduced  the  number  of  program  change  drivers  to  those  that  the  experts 
considered  to  have  moderate  or  high  likelihood  of  change  during  program  execution.  While 
the  matrix  manipulation  techniques  will  often  remove  many  of  the  cycles  in  the  matrix,  expert 
judgment  is  also  required  to  eliminate  cycles  that  are  not  removed  by  the  algorithms  and 
rating  criteria.  In  the  context  of  this  retrospective,  we  also  realized  that  asking  the  experts  to 
mentally  reconstruct  what  potential  changes  might  have  been  considered  at  the  early  stages 
of  the  program  did  not  avoid  problems  in  bias  based  on  later  experience.  But  if  implemented 
at  pre-Milestone  A  as  envisioned,  these  judgments  represent  the  reality  of  the  early  life 
cycle  estimation  process.  In  the  end,  we  were  left  with  30  program  change  drivers  that 
formed  the  acyclic  graph  required  for  the  construction  of  the  BBN. 

In  assigning  the  required  conditional  probabilities  for  the  BBN  to  each  change  driver, 
we  utilized  both  the  experts’  elicited  judgments  of  probability  and  the  ranges  of  variance 
produced  from  the  expert  calibration  experiments  performed  earlier.  The  elicited 
probabilities  were  used  to  directly  populate  some  portions  of  the  BBN.  However,  we  quickly 
realized  that  it  was  not  feasible  to  elicit  all  of  the  probabilities  and  conditional  probabilities 
required  for  such  a  complex  BBN.  Hence,  we  adapted  an  algorithmic  approach  to 
specifying  the  needed  probabilities.  To  represent  the  uncertainty  in  the  elicited  probabilities 
and  to  incorporate  this  into  the  computed  probabilities,  we  used  the  second  element  noted 
earlier,  the  ranges  of  variance  produced  by  experiments  conducted  to  calibrate  expert 
judgment  to  a  90%  confidence  range.  This  calibration  research  is  the  second  thrust  of  this 
work  and  is  documented  in  a  separate  technical  report  (Goldenson  &  Stoddard,  2013). 

For  purposes  of  demonstration,  we  relied  on  using  the  results  of  those  experiments. 
However,  in  a  “live  action”  MDAP,  we  would  use  the  actual  program  experts’  calibration 
results,  which  would  be  obtained  through  a  calibration  test.  The  technical  workshop  with  the 
MDAP  experts  would  then  serve  to  both  elicit  their  required  judgments  as  described  earlier 
and  allow  them  to  participate  in  a  series  of  calibration  training  exercises.  The  exercises 
sharpen  expert  abilities  to  exert  less  overconfident  and  less  overoptimistic  judgment  while 
also  producing  the  required  data  for  us  to  capture  uncertainty  within  the  BBN. 

The  resulting  retrospective  BBN  enabled  the  output  of  probability  distributions  used 
as  inputs  to  the  cost  estimation  tool.  We  constructed  linkages  to  the  SEER-SEM  cost 
estimation  tool  used  by  the  program  for  the  system  software  components  comprising  it. 
Monte  Carlo  techniques  allowed  us  to  generate  confidence  intervals  for  these  distributions, 
which  were  then  used  for  input  to  the  cost  model. 

We  are  close  to  completing  the  retrospective  and  will  be  comparing  the  results  of  the 
QUELCE  model  with  the  estimates  and  actual  costs  produced  by  the  program.  The  conduct 
of  the  retrospective  helped  us  refine  our  elicitation  approach,  demonstrated  the  complexity 
of  populating  a  BBN  at  scale,  and  illuminated  the  need  for  calibrating  teams  of  experts,  not 
just  individuals.  Remaining  work  involves  obtaining  a  review  of  our  decisions  about 
connecting  the  BBN  to  cost  models  such  as  COCOMO  and  SEER. 

Conclusion 

Extensive  cost  overruns  have  been  endemic  in  defense  programs  for  many  years.  A 
significant  part  of  the  problem  is  that  the  information  used  for  cost  estimates  of 
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unprecedented  systems  must  rely  heavily  on  expert  judgments.  When  done  early  in  the 
system’s  life  cycle,  the  estimate  is  based  only  on  the  concept  and  incorporates  much 
uncertainty  as  to  how  that  concept  will  be  developed  into  a  fully  deployed  operational 
system.  QUELCE  aims  to  reduce  the  adverse  effects  of  that  uncertainty.  Important  program 
change  drivers  and  the  dependencies  among  them  that  may  not  otherwise  be  considered  in 
forming  estimates  are  made  explicit  to  improve  their  realism  and  accuracy.  The  basis  of  an 
estimate  is  documented  explicitly,  which  facilitates  updating  the  estimate  during  program 
execution  and  helps  others  to  make  informed  judgments  about  their  accuracy.  Variations  in 
the  range  of  possible  states  of  the  program  change  drivers  that  may  occur  under  different 
likely  scenarios  are  explicitly  considered.  The  use  of  probabilistic  methods  combining 
Bayesian  belief  systems  and  Monte  Carlo  simulation  will  ultimately  place  the  cost  estimates 
within  a  more  realistic  range  of  uncertainty. 
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